Characterizing Network Complexity and Learning Efficiency by the Ratio of Weight Interdependence to Sensitivity

We extend previous research on parameter dynamics of digital filters to examine weight sensitivity and interdependence in feedforward networks. Weight sensitivity refers to the effect of small weight perturbations on the network's output, and weight interdependence refers to the degree of co-linearity between weights. A combined measure of the weight space (τ), defined as the ratio of weight interdependence to sensitivity, is explored in networks with hidden-unit activation functions of different complexity in the contexts of learning (1) a non-linearly separable bivariate normal classification task, (2) the XOR problem, (3) sigmoidal functions, and (4) sine functions. Simulations show that networks with more complex activation functions give rise to a smaller τ and more rapid learning, suggesting that weight sensitivity and interdependence together are indicative of network complexity and are predictive of learning efficiency.